Computing Precision and Recall with Missing or Uncertain Ground Truth
نویسندگان
چکیده
In this paper we present a way to use precision and recall measures in total absence of ground truth. We develop a probabilistic interpretation of both measures and show that, provided a sufficient number of data sources are available, it offers a viable performance measure to compare methods if no ground truth is available. This paper also shows the limitations of the approach, in case a systematic bias is present in all compared methods, but shows that it maintains a very high level of overall coherence and stability. It opens broader perspectives and can be extended to handling partial or unreliable ground truth, as well as levels of prior confidence in the methods it aims to compare.
منابع مشابه
On Performance Characterization and Optimization for Image Retrieval
In content-based image retrieval (CBIR) performance characterization is easily being neglected. A major difficulty lies in the fact that ground truth and the definition of benchmarks are extremely user and application dependent. This paper proposes a two-stage CBIR framework which allows to predict the behavior of the retrieval system as well as to optimize its performance. In particular, it is...
متن کاملAn Iterative and Re-weighting Framework for Rejection and Uncertainty Resolution in Crowdsourcing
In practical applications of crowdsourcing, labelers may be uncertain or refuse to label a particular instance (or reject) due to the inherent difficulty, and each labeler may be given a different set of instances for big dataset applications. These various issues lead to missing and uncertain labels. Existing crowdsourcing methods have limited capabilities when these two problems exist. In thi...
متن کاملGround Truth Energies for Hierarchies of Segmentations
In evaluating a hierarchy of segmentations H of an image by ground truth G, which can be partitions of the space or sets, we look for the optimal partition in H that ”fits” G best. Two energies on partial partitions express the proximity from H to G, and G to H. They derive from a local version of the Hausdorff distance. Then the problem amounts to finding the cut of the hierarchy which minimiz...
متن کاملSMART-INT: A SYSTEM FOR ANSWERING QUERIES OVER WEB DATABASES USING ATTRIBUTE DEPENDENCIES by
Many web databases can be seen as providing partial and overlapping information about entities in the world. To answer queries effectively, it is required to integrate the information about the individual entities that are fragmented over multiple tables. At first blush this is just the inverse of a traditional database normalization problem the universal relation is to be reconstructed from th...
متن کاملPrecision and Recall Without Ground Truth
In this paper we present a way to use precision and recall measures in total absence of ground truth. 1 Precision and Recall 1.1 General Definitions and Notation Precision Pr and Recall Rc (and often associated F-measure or ROC curves) are standard metrics expressing the quality of Information Retrieval methods [8]. They are usually expressed with respect to a query q (or averaged over a series...
متن کامل